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Human  operators  supervising  multiple  uninhabited  air  and  ground  vehicles  (UAV s 
and  UGVs)  under  high  task  load  must  be  supported  appropriately  in  context  by  auto¬ 
mation.  Two  experiments  examined  the  efficacy  of  such  adaptive  automation  in  a 
simulated  high  workload  reconnaissance  mission  involving  four  subtasks:  (a)  UAV 
target  identification;  (b)  UGV  route  planning;  (c)  communications,  with  embedded 
verbal  situation  awareness  probes;  and  (d)  change  detection.  The  results  of  the  first 
“baseline”  experiment  established  the  sensitivity  of  a  change  detection  procedure  to 
transient  and  nontransient  events  in  a  complex,  multi-window,  dynamic  display.  Ex¬ 
periment  1  also  set  appropriate  levels  of  low  and  high  task  load  for  use  in  Experiment 
2,  in  which  three  automation  conditions  were  compared:  manual;  static  automation, 
in  which  an  automated  target  recognition  (ATR)  system  was  provided  for  the  UAV 
task;  and  adaptive  automation,  in  which  individual  operator  change  detection  perfor¬ 
mance  was  assessed  in  real  time  and  used  to  invoke  the  ATR  if  and  only  if  change  de- 
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lection  accuracy  was  below  a  threshold.  Change  detection  accuracy  and  situation 
awareness  were  higher  and  workload  was  lower  for  both  automation  conditions  com¬ 
pared  to  manual  performance.  In  addition,  these  beneficial  effects  on  change  detec¬ 
tion  and  workload  were  significantly  greater  for  adaptive  compared  to  static  automa¬ 
tion.  The  results  point  to  the  efficacy  of  adaptive  automation  for  supporting  the 
human  operator  tasked  with  supervision  of  multiple  uninhabited  vehicles  under  high 
workload  conditions. 


INTRODUCTION 

Uninhabited  vehicles  (UVs)  and  other  robotic  systems  are  being  introduced  in 
rapid  fashion  into  the  military  to  extend  manned  capabilities,  provide  tactical  flexi¬ 
bility,  and  act  as  “force  multipliers”  (Barnes,  Parasuraman,  &  Cosenzo,  2006; 
Cummings  &  Guerlain,  2007).  In  the  U.S.  Army’s  Future  Combat  Systems  (FCS), 
for  example,  battlefield  force  structures  will  be  redesigned  to  be  flexible,  reconfig- 
urable  components  tailored  to  specific  combat  missions.  The  human  operators  of 
these  systems  will  be  involved  in  supervisory  control  of  UVs  with  the  possibility  of 
occasional  manual  intervention.  In  the  extreme  case,  soldiers  will  operate  multiple 
systems  while  on  the  move  and  while  under  enemy  fire.  Because  of  the  consequent 
increase  in  the  cognitive  workload  demands  on  the  soldier,  automation  will  be 
needed  to  support  human-system  performance.  For  example,  automated  decision 
aids  can  allow  tactical  decisions  to  be  made  more  rapidly,  thereby  shortening  the 
“sensor-to-shooter”  loop  in  command  and  control  (C^)  systems  (Adams,  2001; 
Rovira,  McGarry,  &  Parasuraman,  2007). 

An  important  design  issue  is  what  the  level  and  type  of  automation  should  be  for 
effective  support  of  the  operator  in  such  systems  (Parasuraman,  Sheridan,  & 
Wickens,  2000).  Unfortunately,  automated  aids  have  not  always  enhanced  system 
performance,  primarily  due  to  problems  in  their  use  by  human  operators  or  to  un¬ 
anticipated  interactions  with  other  subsystems.  Problems  in  human-automation  in¬ 
teraction  have  included  unbalanced  mental  workload,  reduced  situation  aware¬ 
ness,  decision  biases,  mistrust,  overreliance,  and  complacency  (C.  Billings,  1997; 
Parasuraman  &  Riley,  1997;  Sarter,  Woods,  &  Billings,  1997;  Sheridan  &  Para¬ 
suraman,  2006;  Wiener,  1988). 

Adaptive  automation  has  been  proposed  as  a  solution  to  the  problems  asso¬ 
ciated  with  inflexible  automation  (Inagaki,  2003;  Parasuraman,  2000;  Parasura¬ 
man  &  Miller,  2006;  Scerbo,  2001).  In  this  approach,  information  or  decision  sup¬ 
port  is  not  fixed  at  the  design  stage  but  presented  appropriately  depending  on 
context  in  the  operational  environment.  Context-sensitive  adaptive  automation  is 
initiated  by  the  system  based  on  critical  mission  events,  operator  performance,  or 
physiological  state  (Barnes  et  ah,  2006;  Parasuraman,  Bahri,  Deaton,  Morrison,  & 
Barnes,  1992).  Adaptive  systems  were  proposed  over  20  years  ago  (Hancock, 


272  PARASURAMAN,  COSENZO,  AND  DE  VISSER 


Chignell,  &  Lowenthal,  1985;  Parasuraman  et  al.,  1992;  Rouse,  1988),  but  empiri¬ 
cal  evaluations  of  their  efficacy  have  only  been  conducted  over  the  last  decade — in 
such  domains  as  aviation  (Parasuraman,  Mouloua,  &  Hilbum,  1999),  air  traffic 
management  (Hilbum,  Joma,  Byrne,  &  Parasuraman,  1997;  D.  B.  Kaber  &  Ends- 
ley,  2004),  and  industrial  process  control  (Moray,  Inagaki,  &  Itoh,  2000).  Further¬ 
more,  there  has  been  only  limited  empirical  research  on  adaptive  automation  to 
support  human  supervision  of  robots  or  unmanned  vehicles  (Parasuraman,  Galster, 
Squire,  Furukawa,  &  Miller,  2005;  Wilson  &  Russell,  2007). 

The  method  of  invocation  in  adaptive  systems  is  a  key  issue  (Barnes  et  al., 
2006).  Parasuraman  et  al.  (1992)  reviewed  the  major  invocation  techniques  and  di¬ 
vided  them  into  five  main  categories;  (a)  critical  events;  (b)  operator  performance 
measurement;  (c)  operator  physiological  assessment;  (d)  operator  modeling;  and 
(e)  hybrid  methods  that  combine  one  or  more  of  the  previous  four  methods.  For  ex¬ 
ample,  in  an  aircraft  air  defense  system,  adaptive  automation  based  on  critical 
events  would  invoke  automation  only  when  certain  tactical  environmental  events 
occur,  such  as  the  beginning  of  a  “pop-up”  weapon  delivery  sequence:  this  would 
lead  to  activation  of  all  defensive  measures  of  the  aircraft  (Barnes  &  Grossman, 
1985).  If  the  critical  events  do  not  occur,  the  automation  is  not  invoked.  Hence,  this 
method  is  inherently  flexible  and  adaptive,  because  it  can  be  tied  to  current  tactics 
and  doctrine  during  mission  planning.  This  method  requires  that  the  contingencies 
and  critical  events  are  in  fact  anticipated,  which  may  not  always  be  possible.  An¬ 
other  disadvantage  of  the  method  is  its  insensitivity  to  actual  system  and  human 
operator  performance.  One  potential  way  to  overcome  this  limitation  is  to  measure 
operator  performance  and/or  physiological  activity  in  real  time.  For  example, 
mental  workload  may  be  inferred  from  performance,  physiological,  or  other  mea¬ 
sures  (Byrne  &  Parasuraman,  1996;  Kramer  &  Parasuraman,  2007;  Wilson  &  Rus¬ 
sell,  2003).  The  measures  can  provide  inputs  to  an  adaptive  system  manager 
(which  could  be  rule  or  neural  network  based).  The  output  of  this  system  then  in¬ 
vokes  automation  to  support  or  advise  the  operator  appropriately,  with  the  goal  of 
balancing  workload  at  some  optimum,  moderate  level  (Parasuraman  et  al.,  1999; 
Wilson  &  Russell,  2003). 

In  addition  to  measures  of  workload,  assessment  of  situation  awareness  might 
also  be  useful  in  adaptive  systems  (D.  B.  Kaber  &  Endsley,  2004).  Reduced  situa¬ 
tion  awareness  has  been  identified  as  a  major  contributor  to  poor  performance  in 
search-and-rescue  missions  with  autonomous  robots  (Burke  &  Murphy,  2004; 
Murphy,  2004).  In  particular,  transient  or  dynamic  changes  in  situation  awareness 
might  be  captured  by  probing  the  operator’s  awareness  of  changes  in  the  environ¬ 
ment.  One  such  measure  is  change  detection  performance.  People  often  fail  to  no¬ 
tice  changes  in  visual  displays  when  they  occur  at  the  same  time  as  various  forms 
of  visual  transients  (Simons  &  Ambinder,  2005).  This  “change  blindness”  phe¬ 
nomenon  has  typically  been  demonstrated  for  basic  laboratory  tasks  or  for  staged 
real-world  activities  such  as  two-person  sporting  games  or  face-to-face  social  in- 
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teraction  (Simons  &  Rensink,  2005).  Change  blindness  also  occurs  with  complex 
visual  displays  used  in  various  military  environments.  Durlach  (2004)  exam¬ 
ined  the  effect  of  transient  distractor  events  (i.e.,  window  closing)  on  detection  of 
changes  in  icon  position,  location,  and  size  in  a  map  display.  She  showed  that 
changes  in  icon  position  on  the  map  (e.g.,  indicating  enemy  unit  movement)  were 
vulnerable  to  distractor  events;  detection  performance  decreased  from  79  to  37% 
when  such  distractors  were  present. 

The  goal  of  the  present  research  was  to  assess  the  efficacy  of  adaptive  automa¬ 
tion  on  UV  operator  performance  in  a  simulated  reconnaissance  mission.  Adaptive 
support  was  triggered  based  on  the  operator’s  change  detection  performance.  Mis¬ 
sion  scenarios  involved  supervision  of  multiple  UVs  and  required  multitasking. 
Effects  of  adaptive  automation  on  performance,  S  A,  and  workload  were  examined. 
We  developed  an  in-house  simulation  capability,  the  Robotic  NCO,  designed  to 
isolate  some  of  the  cognitive  requirements  associated  with  a  single  operator  con¬ 
trolling  robotic  assets  within  a  larger  military  environment  (Barnes  et  al.,  2006). 
The  goal  was  to  create  a  microworld  with  face  validity  for  future  military  opera¬ 
tions  involving  UAVs  and  UGVs  while  providing  for  a  degree  of  experimental  con¬ 
trol.  For  example,  the  design  of  the  UGV  task  (described  below)  was  based  on  field 
observations  of  the  Army’s  Experimental  Unmanned  Vehicle  (XUV)  that  currently 
uses  an  autonomous  navigation  system  in  the  manner  simulated  in  our  studies.  The 
simulation  required  the  participant  to  complete  four  interrelated,  military-relevant 
tasks;  (s)  a  UAV  target  identification  task;  (b)  a  UGV  route  planning  task;  (c)  a 
communications  task  with  an  embedded  verbal  situation  awareness  probe  task; 
and  (d)  an  ancillary  task  designed  to  assess  situation  awareness  using  a  probe  de¬ 
tection  method,  a  change  detection  task  embedded  within  a  situation  map.  We  con¬ 
ducted  two  experiments  with  the  Robotic  NCO  simulation.  The  first  was  a  “base¬ 
line”  study,  without  automation,  in  which  we  investigated  change  detection  and 
other  aspects  of  performance  as  a  function  of  parametric  variation  in  task  load.  In  a 
subsequent  main  experiment,  we  examined  the  effects  of  adaptive  automation  on 
performance,  workload,  and  situation  awareness  in  the  same  task  under  conditions 
of  low  and  high  task  load. 

In  the  baseline  experiment  we  varied  task  load  by  manipulating  the  difficulty  of 
the  UAV  and  communications  tasks  at  each  of  two  levels  in  a  2  x  2  factorial  design. 
In  the  embedded  change  detection  task,  an  icon  on  the  situation  map  changed  its 
location  at  unpredictable  times  during  the  simulated  mission.  On  the  basis  of  the 
extensive  change  blindness  literature  (Simons  &  Ambinder,  2005),  we  predicted 
that  change  detection  performance  would  be  especially  poor  if  the  change  oc¬ 
curred  when  a  visual  transient  was  simultaneously  present — in  this  experiment, 
when  the  UGV  stopped  and  requested  assistance  from  the  operator,  in  which  case 
the  UGV  status  bar  flashed.  However,  in  a  complex  visual  display  where  many 
items  compete  for  attention,  change  detection  performance  may  be  poor  even 
without  such  visual  transients,  due  to  the  need  for  attention  to  be  allocated  to  many 
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different  subtasks,  windows,  and  display  locations  (Durlach,  2004;  Parasuraman, 
Barnes,  &  Cosenzo,  2007;  Thomas  &  Wickens,  2006).  We  therefore  predicted  that 
change  detection  accuracy  would  be  low  even  in  the  absence  of  an  explicit  display 
transient,  although  not  as  low  as  with  the  UGV  flash  event.  To  test  this  prediction, 
we  also  included  change  events  when  participants  were  engaged  in  the  UAV  task, 
when  no  explicit  visual  transient  was  present. 


BASELINE  EXPERIMENT 


Methods 

Participants 

Sixteen  young  adults  (9  women,  7  men)  aged  18-26  years  (mean  =  20.5)  partic¬ 
ipated.  The  experiment  lasted  approximately  2  hours  and  participants  were  paid 
$15.00  per  hour. 

Robotic  NCO  Simuiation 

The  Robotic  NCO  simulation  involved  four  interrelated  tasks  that  participants 
had  to  perform  in  order  to  achieve  the  overall  goal  of  reconnaissance:  (a)  a  UAV 
target  identification  task;  (b)  a  UGV  route-planning  task;  (c)  a  communications 
task  with  an  embedded  verbal  situation  awareness  probe  task;  and  (d)  a  change  de¬ 
tection  task.  These  tasks  were  presented  in  separate  windows  of  a  computer  moni¬ 
tor.  At  a  comfortable  viewing  distance  the  display  subtended  about  44°  x  33°  of  vi¬ 
sual  angle.  In  addition,  a  situation  map  showing  the  reconnaissance  area  was 
presented  in  a  separate  window  at  the  bottom  of  the  display  (see  Figure  1  A).  Partic¬ 
ipants  were  trained  to  perform  either  the  UAV  or  the  UGV  tasks  by  switching  be¬ 
tween  the  associated  display  windows  using  the  designated  buttons  when  one  task 
or  the  other  demanded  their  attention.  At  the  same  time,  participants  were  required 
to  respond  as  needed  in  the  communications,  situation  awareness  probe,  and 
change  detection  tasks.  The  simulated  mission  began  with  the  preplanned  flight  of 
the  UAV  and  the  movement  of  the  UGV  over  the  terrain  to  be  searched  and  ended 
when  the  UAV  had  completed  its  flight,  the  UGV  had  completed  its  route,  and  the 
situation  map  was  populated  with  all  identified  targets.  Total  mission  time  was  ap¬ 
proximately  5  minutes. 

UAV  Task.  The  UAV  task  simulated  the  arrival  of  electronic  intelligence  hits 
(“elints”)  from  possible  targets  in  the  terrain  over  which  the  UAV  flew.  When  a  tar¬ 
get  was  detected,  it  was  displayed  in  the  UAV  view  as  a  white  square  in  a  yellow 
circle  (Figure  IB).  Participants  were  told  that  when  a  target  was  presented,  they 
were  to  zoom  in  on  that  location,  which  opened  up  a  window  of  the  UAV  view 
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UAV  status 
menu 


Zoomed-in 
UAV  view  with 
possible  target 


UGV  status 
menu 


Situation  map  with 
UAV  and  UGV 
paths  and  friendly 
units 


1 


Communications 


FIGURE  1 A  Robotic  NCO  simulation  with  the  UAV  task  in  the  zoomed-in  view  mode.  The 
four  subtasks,  UAV  target  identification,  UGV  route-planning,  communications  (with  embed¬ 
ded  verbal  situation  awareness  probes),  and  change  detection  (involving  unit  movement  in  the 
situation  map),  were  displayed  in  separate  windows  as  shown. 


(Figure  1  A).  They  were  required  to  identify  the  target  from  a  list  of  possible  types 
(for  which  they  had  received  prior  training).  The  participant  clicked  the  elint 
marker  (i.e.,  white  squares)  with  the  mouse  to  obtain  a  clear  image  of  the  target.  He 
or  she  then  used  a  right  mouse-click  to  identify  the  target  as  enemy  or  unknown. 
An  “enemy”  was  indicated  by  a  red  triangle,  and  an  “unknown”  was  indicated  by  a 
yellow  triangle.  This  procedure  had  to  be  completed  within  6  seconds  or  the  target 
would  disappear,  resulting  in  a  “missed”  target  by  the  participant.  Once  identified, 
the  target  icon  was  then  displayed  on  the  situation  map.  The  difficulty  of  the  UAV 
targeting  task  was  manipulated  by  varying  the  number  of  elint  targets  to  be  identi¬ 
fied,  as  described  below. 

UG  y  Task.  At  the  same  time  as  the  UAV  continued  on  its  flight  path  over  the 
terrain,  the  UGV  moved  through  the  area  following  a  series  of  preplanned  way- 
points  indicating  areas  of  interest  (AOI).  During  the  mission,  the  UGV  would  stop 
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Non-zoomed-in  UAV  view 
Elint  marker  present 


FIGURE  1  B  The  UAV  task  in  the  non-zoomed-in  view  mode  and  the  situation  map. 


at  various  times,  at  which  point  the  UGV  status  bar  would  flash.  There  were  two 
reasons  why  the  UGV  stopped;  it  had  reached  the  named  AOI  or  its  path  was  by 
blocked  by  an  unknown  obstacle.  The  participant  was  instructed  to  then  switch  to 
the  UGV  display  and  click  on  the  UGV  bar  that  accessed  the  view  (simulating 
video  images)  from  the  UGV  (Figure  1C).  The  UGV  did  not  move  forward  until 
the  participant  selected  an  action,  as  follows.  If  the  UGV  had  reached  an  AOI,  the 
participant  reconnoitered  the  area  and  then  restarted  the  UGV  along  its  preplanned 
path  by  selecting  the  Continue  button.  If  an  obstacle  was  present,  the  UGV  view 
showed  a  picture  of  it.  (The  UGV  view  was  displayed  in  the  same  location  as  the 
UAV  view  on  the  computer  monitor.)  When  the  UGV  encountered  an  obstacle,  it 
was  one  of  two  types,  a  blocking  obstacle  (e.g.,  log,  ditch)  or  a  traversable  obstacle. 
A  blocking  obstacle  required  replanning  and  the  participant  rerouted  the  UGV  by 
selecting  the  Re-route  button.  A  traversable  obstacle  required  that  the  participant 
had  to  resume  the  UGV  along  its  preplanned  path  by  selecting  the  Continue  button. 
The  difficulty  of  the  UGV  monitoring  task  was  kept  at  a  fixed,  relatively  high  level 
by  requiring  the  participant  to  respond  to  a  total  of  seven  UGV  stops  (three  AOIs 
and  four  obstacles,  two  blocking  and  two  traversable). 
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Zoomed-in  UGV 
view  with 

possible  obstacle  UGV  status 


FIGURE  1 C  The  UGV  task  in  zoomed-in  view  showing  possible  obstacle. 


Communications  task  1.  Caii  sign  acknowiedgment.  Participants  re¬ 
ceived  messages  intermittently  while  performing  the  UAV  and  UGV  tasks. 
Communications  were  presented  both  visually  in  a  separate  communications 
window  (see  Figure  lA)  and  acoustically  over  a  speaker.  The  messages  re¬ 
quested  updates  on  the  UGV/UAV  status  and  the  location  of  particular  targets 
(to  assess  situation  awareness,  as  described  further  below).  Participants  also 
had  to  monitor  the  communications  for  their  own  call  sign  (which  they  had 
been  given  previously).  Twenty  call  sign  messages  were  presented  at  random 
intervals  and  had  either  to  be  either  acknowledged  with  a  key  press  if  the  partic¬ 
ipant’s  own  call  sign  was  presented  or,  if  different,  ignored.  Call  sign  acknowl¬ 
edgement  had  to  occur  within  5  seconds  of  the  message  or  else  a  missed  call 
sign  would  result.  The  difficulty  of  the  communications  task  was  manipulated 
by  varying  the  proportion  of  own  call  signs  relative  to  the  total  number  of  call 
signs  presented  (see  below). 
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Communications  task.  2.  Verbai  situation  awareness  probes.  The  com¬ 
munications  task  also  included  embedded  messages  designed  to  assess  the  partici¬ 
pants’  situation  awareness.  The  questions  were  presented  during  the  mission  and 
required  a  “Yes”  or  “No”  response  from  the  participant.  The  questions  spanned 
Types  1  and  2  of  Endsley’s  (1995)  taxonomy  of  situation  awareness — perception 
and  comprehension,  respectively.  Examples  were  (a)  Has  the  UGV  required  re¬ 
routing?  (b)  Are  there  enemy  troops  in  sector  X?  (c)  Is  the  UAV  in  Sector  X  able  to 
find  targets  near  enemy  units?  (d)  Are  there  enemy  troops  within  two  grid  squares 
of  sector  X? 

These  questions  were  presented  without  any  “freezing”  or  stoppage  of  the  pri¬ 
mary  simulated  mission.  Participants  were  instructed  to  respond  to  the  questions 
when  they  occurred  and  then  return  to  performing  the  other  concurrent  tasks. 

Change  detection  task.  In  many  tactical  military  operations,  the  operator’s 
situation  display  may  often  be  updated  without  warning,  so  that  the  operator  may 
miss  the  change.  To  capture  this,  a  change  detection  task  was  embedded  into  the 
situation  map  of  the  Robotic  NCO  display  to  obtain  a  performance  probe-based 
measure  of  situation  awareness.  At  unpredictable  times  during  the  simulated  mis¬ 
sion,  and  after  the  situation  map  had  been  populated  to  a  degree,  an  icon  on  the  sit¬ 
uation  map  (a  target  previously  identified  by  the  participant)  changed  its  location 
by  two  grid  squares  (-traversing  approximately  4°  of  visual  angle).  Participants 
were  instructed  that  such  changes  might  occur  and  that  if  they  noticed  them  to 
press  the  space  bar.  Only  a  simple  detection  response  was  required,  not  identifica¬ 
tion  or  recognition.  Eight  icon  changes  were  presented  at  random  intervals  during 
the  5-minute  mission.  Half  of  the  changes  (four)  occurred  during  one  of  the  UGV 
stops,  when  the  UGV  status  bar  flashed  (transient  event  condition).  The  other  four 
changes  occurred  while  participants  were  engaged  in  the  UAV  task  (nontransient 
event  condition). 

It  is  important  to  note  that  participants  were  told  that  the  UAV,  UGV,  and  verbal 
S  A  communications  tasks  in  this  simulation  were  not  independent  but  coordinated 
tasks  that  supported  the  overall  goal — a  reconnaissance  mission  in  which  partici¬ 
pants  had  to  be  aware  of  friendly  and  enemy  unit  movements  and  of  the  positions 
of  their  UAV  and  UGV  assets.  The  verbal  SA  queries  that  were  posed  over  the 
communications  channel  provided  an  evaluation  of  how  well  participants  followed 
these  instructions.  Eurthermore,  participants  were  told  that  they  would  be  asked  at 
the  end  of  the  mission  to  select  the  best  path  for  a  platoon  to  follow,  so  that  they 
could  not  simply  ignore  UAV  targets  or  enemy  units  once  identified  but  had  to  inte¬ 
grate  them  into  their  overall  map  of  the  battlefield. 

Workioad  and  situation  awareness  questionnaires.  A  subjective  over¬ 
all  rating  was  given  at  the  end  of  each  5-minute  mission  trial  on  participant’s  per¬ 
ceived  overall  workload  (OW)  and  situation  awareness.  This  was  a  single  number 
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from  0  to  100  for  each  rating.  Both  OW  and  situation  awareness  criteria  were 
adapted  from  the  NASA-TLX  (Hart  &  Staveland,  1998)  and  the  Cognitive 
Compatibility  Situation  Awareness  Technique  questionnaire  (CC-S  ART;  Taylor, 
1990). 

Procedure 

Following  familiarization,  training,  and  practice  on  the  Robotic  NCO  simula¬ 
tion,  participants  performed  eight  simulated  missions,  each  lasting  5  minutes 
each.  Participants  were  asked  to  take  the  role  of  a  robotic  operator  in  a  Mounted 
Combat  System  company  (MCS).  They  were  asked  to  conduct  a  reconnaissance 
mission  for  the  MCS  platoon  using  their  UAV  and  UGV  assets.  The  UAV  and 
UGV  starting  point,  ending  point,  and  path  were  preplanned  by  the  experimenter, 
and  except  for  UGV  reroutes,  were  not  under  control  of  the  participant.  The  UAV 
traveled  faster  than  the  UGV  and  provided  surveillance  information  to  the  partic¬ 
ipant  as  described  previously.  The  UGV  followed  its  routed  path  and  when  an 
event  occurred,  waited  for  operator  input,  as  described.  While  supervising  the 
two  robotic  assets  the  operator  received  communications,  either  call  sign  ac¬ 
knowledgments  or  status  queries,  and  performed  the  change  detection  task,  as 
described  previously. 

The  baseline  experiment  was  a  2  x  2  within-subjects  design,  with  the  manipu¬ 
lated  factors  being  the  difficulty  of  the  UAV  and  communications  (call  sign) 
tasks.  For  the  UAV  task,  task  load  or  difficulty  was  manipulated  by  varying  the 
number  of  targets.  There  were  10  targets  in  the  low  condition  and  20  targets  in  the 
high  condition.  Task  difficulty  in  the  communications  task  was  manipulated  by 
varying  the  uncertainty  associated  with  seeing  (hearing)  one’s  own  call  sign.  In 
the  low  uncertainty  (low  difficulty)  condition,  16  of  the  20  call  signs  presented 
were  the  participant’s  own  call  sign;  hence,  participants  had  very  little  uncer¬ 
tainty  as  to  whether  their  call  sign  had  been  presented  while  engaged  in  multiple 
other  tasks.  In  the  high  uncertainty  (high  difficulty)  condition,  only  4  of  the  20 
call  signs  were  the  participant’s  own.  As  a  result,  greater  vigilance  on  the  part  of 
the  participant  was  required  given  the  lower  probability  of  own  call  sign.  The  two 
levels  of  difficulty  on  the  UAV  and  communications  were  factorially  combined 
to  yield  four  scenarios,  each  of  which  was  presented  twice,  resulting  in  eight  total 
missions.  The  order  of  presentation  of  mission  scenarios  was  counterbalanced 
between  participants. 

The  following  dependent  variables  were  measured:  (a)  UAV  target  acquisi¬ 
tion:  Accuracy  (proportion  correct)  and  RT  to  identify  target  as  friendly  or  un¬ 
known.  (b)  UGV  route  planning:  Reaction  time  (RT)  to  implement  new  route 
on  blocking  obstacle  event  or  to  observe  an  AOI  and  continue  UGV.  (c)  Com¬ 
munications:  RT  and  percent  missed  for  own  call  sign  acknowledgments,  (d) 
Change  detection:  Accuracy  (proportion  correct)  and  RT  to  correctly  detected 
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FIGURE  2  Effects  of  visual  transient  and  nontransient  events  on  change  detection  accuracy 
under  conditions  of  low  and  high  UAV  and  communications  task  difficulty. 

changes,  (e)  Situation  Awareness;  Overall  situation  awareness  as  measured  by 
the  CC-SART,  and  mean  number  of  correct  responses  on  the  situation  aware¬ 
ness  questions  presented  in  the  communications  task,  (f)  Overall  workload:  OW 
rating. 


RESULTS 


Multivariate  analyses  of  variance  (MANOVAs)  followed  by  ANOVAs  were  con¬ 
ducted  on  the  data.  The  within-subjects  variable  included  in  the  analyses  were 
communications  task  difficulty  (uncertainty)  and  UAV  task  difficulty.  For  the 
change  detection  performance  analysis,  Change  Type  was  also  included  as  a 
within-subjects  variable. 

UAV  Target  Identification 

There  was  a  significant  effect  of  communications  task  difficulty  on  UAV  target 
identification  performance,  F  (2, 14)  =  8.07,/?  <  .005.  ANOVAs  that  were  run  to  de¬ 
termine  whether  RT  or  accuracy  (percentage  correct)  contributed  to  the  main  effect 
gave  a  significant  effect  for  RT,  F  (1,15)  =  8.66, /?  <  .01.  Surprisingly,  mean  RT  to 
UAV  targets  was  significantly  lower  when  communications  uncertainty  was  high 
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(x  =  1.48  s,  SE  =  .03)  than  when  it  was  low  (x  =  1.56  s,  SE  =  .05).  There  were  no 
other  significant  effects. 

UGV  Task  Performance 

RTs  to  the  UGV  stops  (continue  or  reroute)  were  higher  when  communications  un¬ 
certainty  was  high  than  when  it  was  low,  but  this  effect  was  moderated  by  a  signifi¬ 
cant  interaction  between  UAV  task  and  communications  task  difficulty,  E  (1,15)  = 
5.15,  p  <  .03.  Pair-wise  comparisons  showed  that  mean  RT  was  lower  in  the  high 
UAV/low  communications  uncertainty  condition  (x  =  3.84  s,  SE  =  .36)  than  in  the 
high  UAV/high  communications  uncertainty  condition  (x  =  5.85  s,  SE  =  .56), 

p<  .001. 

Communications  Task  Performance 

There  was  a  significant  effect  for  communications  task  difficulty  on  communica¬ 
tions  task  performance,  E  (1,14)  =  158.7,  p  <  .001.  ANOVAs  showed  a  significant 
effect  only  for  RT,  E  (1,15)  =  331.5,p  <  .01.  Mean  RT  for  own  call  sign  acknowl¬ 
edgment  was  significantly  higher  when  communications  task  uncertainty  was  high 
(x  =  3.42  s,  SE  =  .10)  than  when  it  was  low  (x  =  2.33  s,  SE  =  .08).  There  were  no 
other  significant  effects. 

Change  Detection  Performance 

Overall,  change  detection  accuracy  (figure  2)  was  relatively  poor  but  was  never¬ 
theless  affected  by  the  presence  (UGV  task)  or  absence  (UAV  task)  of  visual  tran¬ 
sients  (Change  Type).  There  was  a  significant  effect  of  Change  Type,  F  (1,15)  = 
13.9,  p  <  .02.  fewer  changes  were  detected  when  there  were  visual  transients  (x  = 
13.3%,  SE  =  4.40)  compared  to  when  transients  were  not  present  (x  =  35.2%,  SE  = 
4.20).  There  was  also  a  significant  interaction  between  Change  Type  and  UAV  task 
difficulty,  U  (1,15)  =  10. 2,  p  <  .001;  for  transient  events,  there  was  no  effect  of  UAV 
Task  difficulty,  possibly  because  accuracy  was  near  floor  (-13%);  for  nontransient 
events,  however,  change  detection  accuracy  was  significantly  lower  under  high 
(28.1%)  than  under  low  (42.2%)  UAV  task  difficulty,  F  (1,15)  =  6.16,  p  <  .02. 

Analyses  of  change  detection  RT  showed  no  significant  effects  of  communica¬ 
tions  task  and  UAV  task  difficulty  or  of  Change  Type. 

Verbal  Situation  Awareness  Probes 

There  were  no  significant  effects  of  level  of  difficulty  for  the  UAV  or  communica¬ 
tions  tasks  on  the  number  of  situation  awareness  questions  answered  correctly.  The 
mean  number  correct  (out  of  4),  irrespective  of  condition,  was  2.67  (SE  =.11) 
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Situation  Awareness  and  Workioad 

Results  for  overall  situation  awareness  showed  a  significant  main  effect  for  the  lev¬ 
els  of  situation  awareness,  F  (2,14)  =  5.98,  p<  .0\,  as  measured  by  the  CC-SART. 
Paired  comparisons  of  the  data  showed  that  the  score  for  level  of  processing  (x  = 
3.68,  SE  =  .36)  was  significantly  lower  than  ease  of  reasoning  (x  =  5.68,  SE  =  .31) 
and  activation  of  knowledge  (x  =  5.43,  SE  =  .30),  p  <  .001.  Ease  of  reasoning  and 
activation  of  knowledge  did  not  differ  significantly. 

Results  for  overall  workload  showed  a  significant  main  effect  for  the  levels  of 
workload,  F  (5,1 1)  =  17.81,  p  <  .001.  Paired  comparisons  of  the  data  showed  that 
mental  demand  was  significantly  higher  than  all  the  other  levels  except  for  effort, 
ps  <  .05.  Eurther,  physical  demand  was  significantly  lower  than  all  the  other  levels 
except  for  frustration,  ps  <  .01 .  Frustration  was  also  significantly  lower  than  all  the 
other  levels,  ps  <  .01.  No  other  comparisons  were  significant. 


DISCUSSION 

The  baseline  experiment  was  designed  to  assess  performance  levels  on  the  Robotic 
NCO  simulation  in  response  to  variation  in  task  load.  The  goal  was  to  determine  an 
appropriate  overall  task  difficulty  level  that  could  be  used  to  assess  effects  of  adap¬ 
tive  automation  in  a  subsequent  main  experiment.  The  baseline  study  was  also  run 
to  determine  whether  change  detection  performance  could  be  sensitively  assessed 
in  the  context  of  a  complex  multitask  simulation.  With  respect  to  the  first  goal,  the 
study  was  successful;  appropriate  conditions  of  task  loading  were  identified.  How¬ 
ever,  the  results  showed  that  some  task  load  manipulations  had  the  expected  effect 
on  performance,  whereas  others  did  not.  For  example,  high  uncertainty  on  the 
communication  task  led  to  longer  RTs  for  own  call  sign  acknowledgment,  as  pre¬ 
dicted.  However,  unexpectedly  the  same  task  difficulty  manipulation  led  to  lower 
RTs  on  the  UAV  target  identification  task,  for  unknown  reasons.  The  effects  of 
UAV  and  communications  task  difficulty  on  performance  on  the  UGV  route-plan¬ 
ning  task  were  also  generally  as  expected,  with  longer  RTs  to  reroute  the  UGVs  un¬ 
der  the  more  difficult  conditions.  Overall,  therefore,  the  results  indicate  that  a  high 
number  of  UAV  targets  and  greater  uncertainty  in  the  communications  task  would 
provide  a  sufficiently  challenging  level  of  task  difficulty  for  use  in  the  main  experi¬ 
ment. 

With  respect  to  the  second  goal  of  the  baseline  experiment,  assessment  of 
change  detection  performance,  the  results  were  encouraging.  Change  detection  ac¬ 
curacy  was  typically  low,  ranging  from  9.4  to  43.8%  across  the  various  conditions 
This  result  indicates  that  the  change  blindness  effect,  which  has  typically  been 
demonstrated  in  simple  laboratory  tasks  or  contrived  social  interactions  (Simons  & 
Ambinder,  2005;  but  see  Durlach,  2004),  also  occurs  with  more  realistic  displays 
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relevant  to  real-world  environments.  Given  that  change  detection  reflects  a  percep¬ 
tual  component  of  situation  awareness,  the  low  values  suggest  a  major  potential 
source  of  system  inefficiency  in  multi-UV  operations.  Despite  the  low  over¬ 
all  level,  however,  change  detection  performance  was,  as  predicted,  poorer  for 
changes  occurring  during  transient  events  (UGV  task)  than  during  nontransient 
events  (UAV  task).  This  indicates  that  the  change  detection  procedure  we  used  was 
sensitive  and  is  encouraging  with  respect  to  the  potential  for  using  it  to  assess  when 
adaptive  automation  might  be  useful  to  support  the  human  operator  supervising 
multiple  UVs.  Another  index  of  the  sensitivity  of  the  change  detection  task  is  that 
accuracy  during  nontransient  events  (UAV  task)  was  reduced  when  the  UAV  task 
difficulty  was  increased  from  low  to  high,  suggesting  that  the  increased  attentional 
demand  associated  with  more  UAV  targets  was  reflected  in  poorer  awareness  of 
changes  in  the  situation  map. 


MAIN  EXPERIMENT 

In  the  main  study  we  examined  the  effects  of  adaptive  automation,  based  on 
real-time  assessment  of  operator  change  detection  performance,  on  performance, 
situation  awareness,  and  workload  in  supervising  multiple  UVs  in  the  Robotic 
NCO  simulation  under  two  levels  of  task  load.  We  used  an  adaptive  automation  in¬ 
vocation  method  first  developed  by  Parasuraman,  Mouloua,  and  Molloy  (1996), 
known  as  performance-based  adaptation.  In  this  method,  individual  operator  per¬ 
formance  (i.e.,  change  detection  performance)  is  assessed  in  real  time  and  used  as 
a  basis  to  invoke  automation.  In  contrast,  in  static  or  model-based  automation,  au¬ 
tomation  is  invoked  at  a  particular  point  in  time  during  the  mission  based  on  the 
model  prediction  that  operator  performance  is  likely  to  be  poor  at  that  time 
(Parasuraman  et  ah,  1992).  This  method  is  by  definition  not  sensitive  to  within-  or 
between-individual  differences  in  performance,  because  it  assumes  that  all  opera¬ 
tors  are  characterized  by  the  model  predictions.  In  performance-based  adaptive  au¬ 
tomation,  on  the  other  hand,  automation  is  invoked  if  and  only  if  the  performance 
of  an  individual  operator  is  below  a  specified  threshold  at  a  particular  point  in  time 
during  the  mission.  If  a  particular  operator  does  not  meet  the  threshold  at  that  time, 
automation  is  invoked.  However,  if  the  threshold  is  exceeded  in  another  operator, 
or  in  the  same  operator  at  a  different  point  in  the  mission,  the  automation  is  not  in¬ 
voked.  Thus,  performance-based  adaptation  is  by  definition  context-sensitive  to  an 
extent. 

To  demonstrate  the  potential  benefit  of  adaptive  automation,  it  must  be  com¬ 
pared  not  only  to  performance  without  automation  (manual  performance)  but  to 
static  automation  (Barnes  et  ah,  2006;  Parasuraman,  1993).  Accordingly,  in  the 
main  experiment,  we  examined  performance  in  the  Robotic  NCO  task  under  three 
conditions:  (a)  manual;  (b)  static  automation,  in  which  participants  were  supported 
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in  the  UAV  task  with  an  automated  target  recognition  (ATR)  system  that  detected 
and  identified  targets;  (c)  adaptive  automation,  in  which  the  ATR  automation  was 
invoked  if  change  detection  performance  was  below  a  threshold,  but  not  otherwise. 
Each  of  these  conditions  was  combined  factorially  with  two  levels  of  task  load,  as 
manipulated  by  variation  in  the  difficulty  of  the  communications  task. 

We  predicted  that  change  detection  performance  and  situation  awareness  as  as¬ 
sessed  using  the  verbal  situation  awareness  probes  would  both  be  enhanced  with 
adaptive  automation,  whereas  overall  mental  workload  would  be  reduced,  follow¬ 
ing  the  logic  of  Parasuraman  et  al.  (1996,  1999).  In  turn  these  benefits  would  be 
greater  for  the  adaptive  compared  to  the  static  automation  condition,  with  both  au¬ 
tomation  conditions  being  superior  to  manual  performance.  Finally,  we  expected 
that  the  selective  benefits  of  adaptive  automation,  if  found,  would  be  greater  under 
high  task  load  than  under  low  task  load. 


METHODS 


Participants 

Sixteen  young  adults  (8  women,  8  men)  aged  18-28  years  (mean  =  21.9)  partici¬ 
pated.  The  experiment  lasted  approximately  2  hours  and  participants  were  paid 
$15.00  per  hour. 

Procedure 

Participants  performed  the  same  Robotic  NCO  simulation  used  in  the  baseline  ex¬ 
periment.  There  was  a  fixed,  high  number  of  UGV  requests  (7)  and  a  fixed,  high 
number  of  UAV  targets  (20).  These  task  parameters  were  combined  with  either  a 
low  (16  own  call  signs  in  20)  or  high  level  of  uncertainty  (4  own  call  signs  in  20)  in 
the  communications  task  to  create  two  conditions  of  overall  task  load,  low  and 
high.  In  addition,  the  change  detection  procedure  of  the  baseline  experiment  was 
used.  Based  on  the  results  of  the  baseline  experiment,  this  combination  of  task  dif¬ 
ficulty  variables  was  considered  to  represent  the  most  challenging  performance 
conditions  and  therefore  the  most  suitable  for  potential  mitigation  through  adap¬ 
tive  automation. 

The  procedure  was  the  same  as  the  baseline  experiment  with  one  exception.  At 
the  beginning  of  the  experiment,  participants  were  told  that  an  automated  target 
recognition  system  might  be  invoked  during  the  missions.  They  were  also  given 
training  on  the  ATR  prior  to  the  main  experimental  blocks.  When  the  automation 
was  about  to  be  invoked,  the  participant  received  a  message  (presented  both 
auditorially  through  the  speaker  and  visually  in  the  communications  window),  in¬ 
dicating  that  the  ATR  was  now  available.  Note  that  the  ATR  automated  the  func¬ 
tion  of  target  identification,  but  participants  still  had  to  attend  to  the  results  of  the 
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automation.  Participants  could  not  simply  ignore  the  UAV  task  when  ATR  was  pro¬ 
vided  because  they  needed  to  integrate  the  information  provided  by  its  sensor  to 
support  the  overall  goal  of  reconnaissance.  As  described  in  the  Methods  section 
previously,  they  were  also  given  SA  queries,  some  of  which  asked  about  the  rela¬ 
tive  position  of  the  UAV  or  of  identified  enemy  units.  Participants  were  also  in¬ 
structed  to  monitor  the  results  of  the  ATR.  Finally,  they  were  required  at  the  end  of 
the  mission  to  evaluate  the  best  platoon  path  to  take  following  the  reconnaissance 
mission.  Hence,  the  ATR  automation  did  not  simply  remove  a  task  but  supported 
the  overall  mission  goal. 

There  were  three  main  automation  conditions.  In  the  Manual  condition,  partici¬ 
pants  performed  all  four  tasks  of  the  Robotic  NCO  simulation  without  automated 
support.  In  the  Static  Automation  conditions  participants  were  supported  in  the 
UAV  task  with  an  ATR  system,  thereby  off-loading  them  of  the  responsibility  of 
identifying  targets  but  not  of  the  responsibility  of  using  the  UAV  sensor  informa¬ 
tion  or  monitoring  its  path  and  evaluating  its  position  with  respect  to  friendly  and 
enemy  units.  The  automation  was  invoked  in  the  middle  (at  ~2.5  min)  of  the  5-min¬ 
ute  mission  (after  four  change  detection  events)  in  all  participants  and  irrespective 
of  the  level  of  performance.  A  similar  procedure  was  used  in  the  Adaptive  Auto¬ 
mation  condition,  except  that  the  Robotic  NCO  software  maintained  a  running 
count  of  an  individual  operator’s  change  detection  accuracy.  After  four  change  de¬ 
tection  events  had  occurred  in  the  middle  (at  ~2.5  minutes)  of  the  5-minute  mis¬ 
sion,  the  current  count  of  detection  accuracy  was  compared  to  a  threshold.  If  the 
threshold  was  not  met,  the  ATR  automation  was  invoked,  but  not  otherwise.  Based 
on  the  results  of  the  baseline  experiment  as  well  as  pilot  work,  a  threshold  of  50% 
accuracy  was  chosen.  It  was  expected  with  this  threshold  that  many  participants 
would  receive  adaptive  aiding,  with  the  proportion  being  higher  under  high  than 
low  task  load.  After  adaptive  automation  invocation  (or  not),  the  mission  contin¬ 
ued  as  before;  that  is,  the  ATR  remained  on  until  the  end  of  the  block. 

The  three  automation  conditions  (Manual,  Static,  Adaptive)  were  combined 
factorially  with  task  load  (low  or  high)  for  six  mission  scenarios  (each  lasting  5 
minutes).  Each  of  these  conditions  was  repeated  (blocks)  in  order  to  see  whether 
any  adaptive  automation  effects  would  be  reduced  with  additional  practice.  The 
experiment  was  therefore  a  2  x  3  x  2  within-subjects  design.  The  within- subjects 
factors  were  the  uncertainty  of  the  communications  task  (low  or  high),  automa¬ 
tion  condition  (Manual,  Static,  and  Adaptive  Automation),  and  block  (one  and 
two). 

Each  participant  completed  twelve  5-minute  mission  blocks.  The  order  of 
blocks  was  counterbalanced.  Eor  statistical  evaluation  of  the  effects  of  automation, 
mission  phase  (pre-automation  invocation  and  post-automation  invocation — in  the 
middle  of  the  5-minute  mission)  was  included  as  a  factor.  Eor  comparability,  even 
though  automation  was  not  invoked  in  the  manual  condition,  performance  mea¬ 
sures  were  calculated  for  each  mission  phase  in  this  condition  as  well. 


286  PARASURAMAN,  COSENZO,  AND  DE  VISSER 


The  same  dependent  variables  as  in  the  baseline  experiment  were  used,  with  the 
exception  that  the  CC-SART  was  dropped  and  only  the  verbal  situation  awareness 
probes  were  used.  Of  the  four  verbal  situation  awareness  questions,  two  were  given 
in  each  half  of  every  mission.  Because  of  the  low  number  of  questions  in  each  half, 
only  an  overall  verbal  situation  awareness  accuracy  score  was  computed. 


RESULTS 

Because  the  main  hypotheses  that  were  tested  in  this  study  involved  the  change  de¬ 
tection  measure,  we  report  the  results  for  this  dependent  variable  first,  followed  by 
performance  on  the  UAV,  UGV,  and  communications  tasks  and  the  subjective 
ratings. 

MANOVAs  and  ANOVAs  were  conducted  for  the  data.  The  within-subjects 
variables  included  in  the  statistical  analyses  were  communications  task  difficulty, 
mission  phase  (pre-post  automation),  automation  condition,  and  block.  Because 
there  were  no  significant  interactions  involving  the  block  factor,  the  data  were  col¬ 
lapsed  across  this  factor. 

Change  Detection  Performance 

The  adaptive  ATR  was  not  invoked  in  three  participants  in  the  low  task  load/adap- 
tive  automation  condition  when  it  was  first  performed  (block  1).  The  ATR  was  also 
not  invoked  in  one  participant  in  the  second  block  of  this  condition.  In  the  high  task 
load/adaptive  automation  condition,  the  ATR  was  invoked  in  all  participants.  The 
results  of  the  statistical  analyses  were  the  same  whether  these  participants  were  in¬ 
cluded  or  excluded  in  the  data  set.  Therefore,  we  first  present  results  with  the  data 
from  all  the  participants,  irrespective  of  whether  the  ATR  was  invoked  adaptively 
for  them  in  the  adaptive  automation  condition. 

Results  for  the  change  detection  accuracy  scores  revealed  significant  main  ef¬ 
fects  for  automation  condition,  F  (2,  30)  =  22.8,  p  <  .00 1 ;  mission  phase,  F  ( 1 ,  15) 
=  151.3,p  <.  001;  and  communications  task  difficulty,  F(l,  15)  =  37.5,  p<  .001.  In 
addition,  the  interaction  of  mission  phase  and  automation  condition  was  signifi¬ 
cant,  F  (2,30)  =  22.1,  p  <  .001.  To  examine  the  interaction  further,  separate 
ANOVAs  were  computed  for  both  the  pre-automation  invocation  and  post-invoca¬ 
tion  phases.  In  the  pre-invocation  phase  before  automation  was  implemented,  there 
were  no  significant  differences  in  change  detection  accuracy  between  conditions. 
However,  the  effect  of  automation  condition  was  significant  for  the  post-invoca¬ 
tion  phase,  F  (2,30)  =  31.3,  p  <.  .001.  As  shown  in  Figure  3,  and  as  verified  by 
pair-wise  comparisons,  participants  detected  significantly  more  icon  changes  in 
the  situation  map  in  the  static  and  adaptive  automation  conditions  compared  to  the 
manual  condition,  ps  <  .001.  Furthermore,  change  detection  accuracy  was  signifi- 
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FIGURE  3  Effects  of  static  and  adaptive  automation  on  change  detection  accuracy,  compared 
to  manual  performance.  Values  are  shown  for  the  pre-automation  invocation  (Pre)  and  post-au¬ 
tomation  invocation  (Post)  mission  phases. 


cantly  higher  in  the  adaptive  automation  condition  than  in  the  static  automation 
condition,  p  <  .02.  Finally,  more  changes  were  detected  when  the  uncertainty  of  the 
communications  was  low  (x  =  36.0%,  SE  =  1.22)  than  high  (x  =  27.2%,  SE  =  .94). 
However,  although  we  predicted  a  greater  effect  of  adaptive  automation  on  change 
detection  accuracy  under  high  than  low  task  load,  the  Automation  Condition  x 
Communications  Task  difficulty  interaction,  F  (2,  30)  =  1 .06,  and  the  Automation 
Condition  x  Mission  Phase  x  Communications  Task  difficulty  interaction,  F  (2, 
30)  =  2.24,  were  not  significant. 

Overall,  there  was  a  marked  improvement  in  change  detection  accuracy  with 
automation,  from  a  mean  of  22.6%  in  the  manual  condition  to  a  mean  of  50.3%  in 
the  two  automation  conditions,  a  1 12%  improvement.  More  importantly,  there  was 
a  54%  improvement  in  change  detection  performance  specifically  associated  with 
adaptive  automation,  from  an  accuracy  of  42.9%  in  the  static  condition  to  57.8%  in 
the  adaptive  condition. 

We  also  compared  change  detection  performance  in  those  participants  for 
whom  automation  was  not  invoked  adaptively  with  those  in  whom  it  was.  Recall 
that  the  ATR  was  not  invoked  adaptively  in  three  participants  in  the  low  task  load 
condition  because  they  exceeded  the  threshold  (see  Eigure  4),  whereas  it  was  in  13 
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F I G  U  R  E  4  Change  detection  accuracy  in  participants  for  whom  automation  was  not  invoked 
adaptively  (No  ATR)  and  in  those  for  whom  it  was  triggered  (ATR),  for  pre-  and  post-invocation 
periods.  The  mean  performance  levels  of  the  same  participants  (No  ATR  and  ATR)  in  the  man¬ 
ual  and  static  automation  conditions  are  also  shown. 


Others.  Figure  4  shows  the  change  detection  performance  of  these  two  groups  in 
the  pre-  and  post-invocation  mission  phases.  By  definition,  the  non- ATR  group  had 
superior  change  detection  accuracy  to  the  ATR  group  in  the  pre-invocation  phase, 
as  confirmed  by  a  f-test  adjusted  using  Levene’s  test  for  unequal  variances  (due  to 
unequal  group  sample  sizes),  t  =  4.26,  p<  .05.  In  the  post-invocation  phase,  how¬ 
ever,  the  two  groups  did  not  differ  significantly,  f  =  .  1 ,  p  >  .  8.  Thus,  adaptive  auto¬ 
mation  had  the  effect  of  “bringing  up”  the  performance  of  the  1 3  participants  to  the 
level  of  the  three  for  whom  adaptation  was  not  required  at  that  point  in  the  mission. 
Figure  4  also  shows  the  mean  performance  levels  of  the  same  non- ATR  and  ATR 
participants  in  the  manual  and  static  automation  conditions.  This  shows  the  same 
pattern  of  results  as  in  Figure  3  where  participants  were  not  subgrouped  in  this 
manner;  both  static  and  adaptive  automation  led  to  superior  performance  com¬ 
pared  to  manual  performance,  with  the  adaptive  automation  condition  leading  to  a 
further  improvement. 
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UAV  Target  Identification 

Performance  on  the  UAV  target  identification  task  was  analyzed  only  for  the  first 
half  of  each  mission  phase,  because  this  function  was  automated  in  the  second  half. 
There  were  no  significant  differences  in  target  identification  accuracy  or  RT  be¬ 
tween  the  manual,  static  automation,  or  adaptive  automation  conditions.  There  was 
also  no  significant  effect  of  communications  task  difficulty  on  UAV  target  identifi¬ 
cation  performance  in  the  pre-invocation  phase. 

UGV  Route  Planning 

There  were  no  significant  effects  or  interactions  of  automation  condition  or  mis¬ 
sion  phase  on  performance  of  the  UGV  route  planning  task.  There  was  a  signifi¬ 
cant  main  effect  for  communications  task  difficulty,  T  ( 1 ,  15)  =  9.57,  p  <  .007.  Re¬ 
sults  showed  that  RTs  to  the  UGV  stops  (continue  or  reroute)  were  higher  when 
communications  uncertainty  was  high  (x  =  4.39  s,  SE  =  0.14)  than  low  (x  =  4.26  s, 
SE  =  0.42). 


Communications  Task 

The  main  effects  of  automation  condition,  F  (4,12)  =  10.5,  p  <  .001;  phase,  F 
(2,14)  =  32.4,  p  <  .001;  and  communications  task  difficulty,  F  (2,14)  =  83.7,  p  < 
.001,  were  significant  for  communications  performance  (accuracy  and  RT).  In  ad¬ 
dition  the  Automation  Condition  x  Phase  interaction  was  significant,  F  (4,12)  = 
16.3,p  <  .001.  All  other  interactions  and  main  effects  were  significant,  ps  <  .01.  To 
determine  whether  RT  or  percentage  correct  contributed  to  the  significant  interac¬ 
tions  and  main  effects,  ANOVAs  were  run.  Results  revealed  that  percentage  cor¬ 
rect  and  RT  contributed  significantly  to  the  main  effect  of  communications  task 
difficulty,  E  (1,15)  =  51.8,p  <  .000  andE  (1,15)  =  70.39,  p  <  .001,  respectively.  A 
greater  percentage  of  call  signs  were  responded  to  and  with  greater  speed  (i.e., 
lower  RT)  in  the  low  uncertainty  communications  condition  (correct:  x  =  86.9%, 
SE  =  1 .29;  RT:  x  =  1 .29  s,  SE  =  .01)  than  the  high  uncertainty  communications  con¬ 
dition  (correct:  x  =  65.3%,  SE  =  1.67;  RT:  x  =  1.29  s,  SE  =  .02). 

ANOVAs  also  showed  a  significant  interaction  of  automation  condition  and 
phase  for  percent  of  call  signs  acknowledged,  E  (2,30)  =  7.07,  p  <  .003.  The  inter¬ 
action  indicates  that  though  there  was  no  significant  difference  between  conditions 
in  the  pre-invocation  phase,  E  <  1.0,  there  was  a  significant  difference  in  the 
post-invocation  phase  E  (2,30)  =  20.9,  p  <  .001.  As  Eigure  5  shows,  and  as 
pair-wise  comparisons  confirmed,  more  call  signs  were  responded  to  correctly  in 
the  static  and  adaptive  automation  conditions  than  in  the  manual  condition,  ps  < 
.003.  Eurthermore,  more  call  signs  were  responded  to  in  the  adaptive  condition 
than  the  static  condition,  p  <.  008. 
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FIGURE  5  Effects  of  static  and  adaptive  automation  on  communications  task  accuracy,  com¬ 
pared  to  manual  performance,  for  pre-  and  post-invocation  periods. 


ANOVAs  showed  a  significant  interaction  of  automation  condition  and  phase 
difficulty  for  RT  to  call  signs,  F  (2,30)  =  19.55,  /?  <  .001 .  RT  did  not  differ  between 
conditions  in  the  pre-invocation  phase,  F  <  1.0,  but  did  in  the  post-invocation 
phase,  F  (2,30)  =  23.1,  /?  <  .001.  Call  signs  were  responded  to  more  quickly  (i.e., 
lower  RTs)  in  the  adaptive  condition  (x  =  1 .27  s,  SE  =  .04)  than  the  static  (x  =  1 .48 
s,  SE  =  .02)  or  manual  (x=  1.51  s,  SE=  .02)  condition,  ps  <  .001.  There  was  no  sig¬ 
nificant  difference  for  RTs  between  the  manual  and  static  automation  conditions. 

Situational  Awareness  Probes 

There  were  significant  main  effects  of  phase,  F  (1,15)  =  18.1,p  =  .001,  and  auto¬ 
mation  condition,  F  (2,30)  =  12.87,  p  <  .001,  and  a  significant  interaction  of  auto¬ 
mation  condition  and  phase,  F  (2,30)  =  9.95,  p  =  .001  for  the  percentage  of  situa¬ 
tion  awareness  questions  correctly  answered.  To  examine  the  interaction  further, 
ANOVAs  were  conducted  for  each  level  of  phase.  There  was  no  significant  differ¬ 
ence  in  answer  accuracy  between  conditions  prior  to  automation  invocation.  There 
was  a  significant  difference  between  automation  conditions  after  automation  invo¬ 
cation,  F  (2,30)  =  19.9,  p  <  .001.  Pair-wise  comparisons  showed  that  showed  that 
more  situation  awareness  questions  were  answered  correctly  in  the  static  (x  = 
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58.2%,  SE  =  5.06)  and  adaptive  automation  (x  =  65.62,  SE  =  4.03)  conditions  than 
in  the  manual  condition  (x  =  29.7%,  SE  =  4.08),  ps  <  .001  (see  Eigure  6).  The  dif¬ 
ference  between  the  adaptive  and  static  automation  condition  was  not  significant. 

Overall  Workload 

There  was  a  significant  main  effect  for  automation  condition  for  subjective  ratings 
of  overall  workload,  F  (2,30)  =  23.5,  p  <  .001 .  No  other  main  effects  or  interactions 
were  significant.  Reported  workload  was  lowest  in  the  adaptive  automation  condi¬ 
tion  (x  =  23.8,  SE  =  1 .07),  intermediate  in  the  static  automation  condition  (x  =  32.5, 
SE  =  2.28),  and  highest  in  the  manual  condition  (x  =  42.5,  SE  =  2.78),  with  all 
pair-wise  comparisons  significant,  ps  <  .001. 

Relationships  Between  Change  Detection,  Situation 
Awareness,  and  Workload 

The  interrelationships  between  change  detection,  situation  awareness,  and  work¬ 
load  are  shown  in  Eigure  7,  which  plots  mean  values  for  these  measures  in  the 
post-invocation  phase  as  a  function  of  automation  condition.  Automation,  and 


Manual  Static  Adaptive 

Automation  Condition 


FIGURE  6  Effects  of  static  and  automation  on  accuracy  of  verbal  situation  awareness  probes, 
compared  to  manual  performance,  for  pre-  and  post-invocation  periods. 
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FIGURE  7  Interrelationships  between  effects  of  static  and  adaptive  automation  on  change 
detection,  situation  awareness,  and  workload.  The  change  detection  and  situation  awareness 
measures  are  indexed  by  percentage  coiTect,  whereas  workload  was  indexed  by  a  subjective  rat¬ 
ing  from  1  to  100. 


adaptive  automation  specifically,  increased  change  detection  accuracy  and  re¬ 
duced  subjective  workload,  as  indicated  by  the  reciprocal  relationship  between 
these  measures  illustrated  in  Figure  7.  In  addition,  automation  also  enhanced  sit¬ 
uation  awareness,  but  though  the  mean  situation  awareness  scores  were  greater 
for  adaptive  compared  to  static  automation,  these  values  were  not  statistically 
different. 


DISCUSSION 

High  cognitive  workload  demands  on  personnel  in  military  systems  working 
with  multiple  UVs  has  mandated  the  use  of  automation  support  (Barnes  et  ah, 
2006).  Because  automation  does  not  always  achieve  the  goal  of  supporting  the 
operator  effectively  (C.  Billings,  1997;  Parasuraman  &  Riley,  1997;  Sarter  et  ah, 
1997),  providing  context-appropriate  aiding — adaptive  automation — ^has  been  pro¬ 
posed  (Miller  &  Parasuraman,  2007;  Parasuraman  et  ah,  1992;  Scerbo,  2001).  In 
the  present  study,  adaptive  aiding  was  provided  to  participants  supervising  multi- 
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pie  UVs,  based  on  real-time  assessment  of  their  change  detection  accuracy.  We 
compared  the  effects  of  manual  performance,  static  automation,  and  adaptive  auto¬ 
mation  on  workload,  situation  awareness,  and  other  aspects  of  task  performance. 

We  predicted  that  change  detection  performance  and  situation  awareness  would 
be  enhanced  with  adaptive  automation,  whereas  overall  mental  workload  would  be 
reduced,  and  that  these  benefits  would  be  specifically  associated  with  adaptive  (as 
opposed  to  static)  automation.  The  results  supported  this  prediction.  Compared  to 
manual  performance,  both  static  and  adaptive  automation  led  to  an  increase  in 
change  detection  accuracy  and  situation  awareness,  whereas  workload  was  re¬ 
duced.  In  addition,  in  comparison  to  static  automation,  there  was  a  further  increase 
in  change  detection  accuracy  and  concomitant  reduction  in  workload  with  adaptive 
automation.  This  last  finding  is  important,  because  simply  demonstrating  perfor¬ 
mance  benefits  due  to  automation  is  insufficient;  rather,  the  specific  benefit,  if  any, 
of  adaptive  automation  must  be  shown,  over  and  above  that  associated  with  static 
automation  (Barnes  et  al.,  2006;  Parasuraman,  1993).  The  results  thus  add  to  the 
growing  literature  pointing  to  the  efficacy  of  adaptive  automation  for  reducing  or 
balancing  operator  workload  and  enhancing  performance  (Inagaki,  2003;  D. 
Kaber  &  Riley,  1999;  Parasuraman  et  al.,  1999;  Scerbo,  2001)  and  confirm  that 
these  benefits  also  accrue  in  the  domain  of  human  operator  supervision  of  multiple 
UVs  in  a  simulated  tactical  reconnaissance  mission. 

There  was  a  reciprocal  relationship  between  the  different  operator  performance 
measures  in  terms  of  the  effects  of  adaptive  automation.  Specifically,  static  auto¬ 
mation  led  to  an  increase  in  both  change  detection  accuracy  and  situation  aware¬ 
ness  and  a  decrease  in  workload,  with  a  further  increase  and  decrease  in  these  mea¬ 
sures  with  adaptive  automation  (see  Figure  7).  However,  it  should  be  noted  that  the 
additional  increase  in  situation  awareness  with  adaptive  automation  was  not  statis¬ 
tically  significant.  Several  studies  have  documented  benefits  of  adaptive  automa¬ 
tion  for  situation  awareness  (D.  B.  Kaber  &  Endsley,  2004).  It  is  possible  that  the 
nonsignificant  trend  we  found  might  simply  reflect  the  relative  insensitivity  of  our 
verbal  probe  measure,  because  we  provided  only  two  such  probes  in  the  post-invo¬ 
cation  phase  of  each  mission  during  which  automatic  target  recognition  was  imple¬ 
mented.  Nonetheless,  the  substantial  benefits  for  change  detection  and  overall 
workload — a  34%  enhancement  in  the  case  of  the  former — argue  strongly  for  the 
efficacy  of  adaptive  automation.  The  reduction  in  overall  workload  was  also  re¬ 
flected  in  better  performance  of  one  of  the  other  three  subtasks  that  participants 
performed,  the  communications  task.  Accuracy  in  responding  to  communications 
was  higher  with  static  automation  than  with  manual  performance,  and  higher  still 
with  adaptive  automation.  Thus  adaptive  automation  was  successful  not  only  in 
supporting  the  human  operator  in  an  appropriate  context — when  their  change  de¬ 
tection  performance  was  low,  pointing  to  low  perceptual  awareness  of  the  evolving 
mission  elements — but  also  freed  up  sufficient  attentional  resources  to  benefit  per¬ 
formance  on  other  less  critical  but  important  subtasks. 
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The  specific  advantage  that  adaptive  automation  based  on  assessment  of  indi¬ 
vidual  performance  or  physiology  brings  is  its  sensitivity  to  within-  and  be- 
tween-individual  differences,  whereas  other  approaches  such  as  model-based 
adaptive  automation  are  not  (Parasuraman  et  al.,  1992).  This  benefit  was  appar¬ 
ent  when  we  compared  the  participants  for  whom  automation  was  not  invoked 
adaptively  in  the  low  task  load  condition  to  those  for  which  automation  was  trig¬ 
gered.  We  found  that  though  change  detection  performance  was  initially  higher 
in  the  former  group  (the  criterion  for  not  invoking  automation),  both  groups  had 
equivalent  levels  of  change  detection  accuracy  in  the  post-invocation  mission 
phase.  Thus,  in  comparison  to  static  automation,  adaptive  automation,  by  provid¬ 
ing  aid  only  in  certain  individuals  when  they  need  it,  or  in  the  same  individual 
when  he  or  she  needs  it  at  different  times,  has  the  effect  of  “leveling”  perfor¬ 
mance  between  and  within  individuals,  thus  providing  for  more  stable  system 
performance. 

What  are  the  practical  implications  of  the  research  reported  in  this  article?  This 
work  is  part  of  a  broader  Army  science  and  technology  program  aimed  at  under¬ 
standing  the  performance  requirements  for  human-robot  interaction  in  future  bat¬ 
tlefields  (Barnes  et  al.,  2006).  Initial  findings  from  this  project  indicate  that  the  pri¬ 
mary  tasks  that  soldiers  are  required  to  perform  place  severe  limits  on  their  ability 
to  monitor  and  supervise  even  a  single  UV,  let  alone  multiple  UVs.  For  example, 
Chen,  Durlach,  Sloan,  and  Bowens  (2008)  examined  target  detection  accuracy  in 
participants  given  either  a  single  robotic  asset  (either  a  UAV,  a  UGV,  or  a 
teleoperated  UGV)  or  all  three  assets.  Target  detection  performance  was  lower 
with  three  than  with  a  single  UV,  and  participants  were  also  less  likely  to  com¬ 
plete  their  missions  in  the  allotted  time.  In  addition,  crew  safety  may  be  compro¬ 
mised  because  soldiers  who  have  to  carry  out  routine  tasks  such  as  radio  commu¬ 
nications  and  ensuring  local  security  also  have  to  supervise  and  manage  several 
robotic  tasks  during  high  workload  mission  segments  (Chen  &  Joyner,  2009; 
Mitchell  &  Henthorn,  2005).  Adaptive  automation  would  therefore  be  particu¬ 
larly  well  suited  to  these  situations  because  of  the  uneven  workload  and  the  re¬ 
quirement  to  maintain  S  A  for  the  primary  as  well  as  the  robotic  tasks.  Our  results 
point  to  the  efficacy  of  adaptive  automation  for  supporting  the  human  operator 
under  these  conditions. 
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